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(  I  1  ABSTRACT 


This  project  addresses  the  questions  of  the  near-term 
utilization  of  color  by  existing  state-of-the-art  binary  vision 
systems.  Using  the  experimental  testbed,  it  was  demonstrated 
that  under  suitable  thresholding  and  filtering,  the  spectral 
contents  of  color  scenes  could  be  artificially  controlled. 
Rejection  of  unwanted  colors  or  background  clutter  could  be 
performed.  Under  single  threshold  selection,  discriminations  of 
objects  identical  in  shapes  and  sizes  but  different  in  colors 
could  be  achieved  by  utilizing  the  luminance  effect  of  colors. 
Color  encoding  combines  the  description  of  colors  and 
thresholding  and  provides  a  simple  interface  to  computer 
processing . 

The  introduction  of  color  discriminations  for  current  machine 
vision  processing  can  significantly  extend  their  range  of 
applications.  Experiments  using  color  filters  to  separate  the 
natural  image  into  color  component  images  show  that  binary  image 
processing  is  a  valuable  next  step  in  color  region  recognition. 


(11]  TECHNICAL  DISCUSSION 
( 1 1 -A ]  INTRODUCTION 


Color-perception  is  an  important  human  ability  and  it  exerts 
profound  influences  on  industry,  business,  science,  engineering, 
as  well  as  art  and  design.  It  is  also  one  of  the  strongest 
visual  cues  (others  are  edges,  grey-scale,  depth  and  shades, 
etc.)  that  we  use  to  distinguish  far/near  objects,  and  probably 
the  only  cue  to  distinguish  colored  objects  with  same  shape, 
size,  and  features.  'Color-perception  can  be  considered  a  very 
individual  experience.  It  is  affected  by  the  physical 
characteristics  of  the  source  and  the  transmitting  medium,  by  the 
physiology  of  the  retina  and  the  visual  nervous  system,  and  also 
by  the  psychological  state  of  processes  of  the  cerebral  cortex  in 
interpreting  the  signals  sent  through  the  retinocortical  pathway. 
The  retinex  (retina-and-cortex )  theory  [8]  of  human  color  vision 
basically  treats  a  color  as  a  three-part  report  from  the  retina, 
independent  of  the  flux  of  radiant  energy.  The  human  eye  has 
evolved  to  see  the  world  in  unchanging  colors,  regardless  of 
shifting,  unpredictable  and  uneven  illumination.  This 
"insensitivity"  to  changing  lighting  makes  color  vision 
attractive  for  industrial  applications. 

Color  vision  is  particularly  useful  in  industrial  automation 
to  recognize,  search,  sort,  and  manipulate  colored  parts  or 
color-coded  objects.  The  use  of  color  permits  part  discrimination 
where  grey-scale  information  alone  is  insufficient  and  often 
times  avoids  the  complicated  and  time-consuming  grey-scale 
analysis  by  using  thresholding.  The  purpose  of  this  research  is 
to  demonstrate  that  color,  together  with  position,  size,  and 
shape  informations  can  be  extracted  from  a  scene  viewed  through  a 
TV  camera.  Commercial  (black  and  white)  binary  vision  systems 
have  begun  to  substantially  affect  productivity  in  automated 
manufacturing.  iheir  value  results  from  their  programmed  ability 
to  locate,  measure,  and  identify  significant  visual 
characteristics  of  manufactured  objects  or  processes.  At  present, 
these  systems  work  on  data  which  have  been  binarized  through  the 
careful  control  of  threshold  and  lighting  in  order  to  provide 
high-contrast  images.  This  has  given  rise  to  an  image  processing 
technology  based  on  the  extraction  of  silhouettes  of  workpieces. 
Such  silhouettes  are  sufficient  to  accomplish  inspection, 
recognition,  and  mensuration  and  to  determine  part  position  and 
orientation . 

The  study  of  machine  binary  color  vision  must  deal  with 
acquisition  of  colors,  differentiation  of  colors,  filtering  and 
selective  processing  of  colors,  measurements  of  colors 
(colorimetry),  and  together  with  all  the  features  (such  as  size, 
shape,  and  position  etc.)  that  a  binary  vision  system  can  extract 
from  a  scene. 

Research  in  the  digital  processing  of  color  images  is  sparse. 
Remote  sensing  has  concerned  itself  with  the  estimation  of  pixel 


statistics  using  multispectral  satellite  data.  For  a  review  of 
this  literature,  see  [12].  There  has  been  recent  interest  in 
color  among  computer  vision  researchers [ 7 ],[ 10 ],[ 13  ]  .  Deserving 
particular  mention  is  the  work  of  Ohta  et.  al.[14].  Using 
registered  red,  green  and  blue  images,  they  found  that  linear 
combination  of  these  images  were  easier  to  segment  than  the  pure 
color  intensity  images.  Several  studies [ 17 ][ 18 ]  of  machine  color 
vision  for  industrial  applications  have  been  reported.  One 
example  was  the  research  done  by  Ito  ( 1975 ) [ 5  ]  , [ 1 1  ]  .  He  utilized 
color  information  in  an  inspection  system  for  IC  mask  patterns. 
Patterns  of  IC  masks  •  were  illuminated  by  red,  green,  and  blue 
light,  respectively.  Defects  of  these  mask  patterns  could  be 
identified  by  the  optical  compositions  of  the  patterns  under 
illumination  of  the  primary  color  light  sources.  Loughlin 
(1982)[9]  reported  Inspectrum  -  a  full  color  inspection  system 
which  checks  the  shape  of  an  industrial  part  and  provides 
detailed  qualitative  inspection  of  surface  color.  Basically,  this 
system  permits  part  discrimination  through  decoding  the  PAL  video 
signal,  digitizing  the  separated  R,  G,  and  B  components  and 
storing  the  data  in  buffer  memory  for  later  comparison.  Position 
information  can  be  obtained  through  the  "inspection  points"  under 
joystick  software  control.  One  disadvantage  of  this  approach  is 
that  position  information  is  not  acquired  automatically  but 
requires  human  interaction.  Fiorini  (1982)  [2]  reported  a  color 
sensing  system  which  functions  as  an  intelligent  peripheral  unit 
with  a  supervisory  computer  which  can  program  up  to  256 
saturation  levels  for  each  chromatic  components:  red,  green,  and 
blue.  This  peripheral  unit  sends  back  three  bits,  one  for  each 
component,  flagging  overflow  or  underflow  of  programmed 
saturation  levels  by  the  chromatic  contents  of  the  sensed  object. 
This  system  does  not  provides  the  size,  shape,  and  position 
information  about  the  inspected  parts.  It  is  capable  of  sensing 
moving  objects  and  is  insensitive  to  lighting  change  in  the 
environment . 


[  1 1 -B ]  RESEARCH  OBJECTIVES 


The  investigation  of  binary  machine  color  vision  was  divided 
into  different  phases  to  attain  the  following  objectives: 

(1)  assembly  of  a  color-imaging  environment  using  Ml  VS1C0  as  the 
major  component. 

(2)  demonstration  of  color-encoding  scheme  for  binary  patterns. 

(3)  methods  for  recombinations  of  binarized  color  images. 

(4)  demonstrations  of  use  of  color  and  shape  for  object 
recognition . 

The  following  subtasks  were  selected  to  achieve  the  above 
goals  : 

*  definition  and  selection  of  color  filters  covering  the 
visible  spectrum. 

*  characterization  of  transmittance  curves  for  color  additive 
filters,  color  subtractive  filters,  and  color  temperature 
compensation  filters  by  using  instruments  such  as  monochromators. 

*  character i zation  of  spectral  distribution  curves  of  existing 
light  sources  including  tungsten  halogen  lamps,  fluorescent 
tubes,  ultra-violet  and  short-wave  tubes,  and  infra-red  long  wave 
lamps  . 

*  assembly  of  light  source,  color  filter  sets  and  wheel 
holder,  and  MI  VS-100  experimental  test-bed. 

*  investigation  of  proper  lighting  for  color  machine  vision, 
control  of  reflected  light  and  emitted  light,  and  selection  of 
opaque,  transparent,  glossy  and  matte  objects  for  study. 

*  experiments  with  monochromatic  objects  and  singling  out 
variables  which  affect  recognition. 

*  tricolor  coding  technique  for  color  objects. 

recog ni tion 


images  for 


*  use  of  tricolor  coding  for  color  objects 
covering  visible?  spectrum. 

*  logical  combination  of  multiple  narrow-band 
specific  discrimination  tasks. 


[II -C 1  BACKGROUND  OF  BINARY  AND  COLOR  VISION 
VS-100  System 


The  VS-100  vision  system  (See 
binary  vision  system.  It  receives 
a  solid-state  or  vidicon  camera 
binary ( black/whi te )  image  that  is 


Figure  1.)  is  a  commercial 
a  video  grey-scale  image  from 
and  thresholds  it  into  a 
run-length  encoded  for  data 


compression  and  subsequent  processing.  Computer  algorithms 
perform  a  connectivity  analysis  of  the  encoded  images,  building 
data  structures  that  represent  essential  features  of  each 
contiguous  region.  The  vision  system  characterizes  blobs  on  the 
basis  of  distinguishing  features  such  as  area,  perimeter,  minimum 
and  maximum  radii,  and  number  of  holes  (See  Figure  3.).  The 
system  can  be  trained  to  analyze  new  objects 
them  to  the  system.  Object  recognition  is 
nearest-neighborhood  classifier  operating  on 
set  of  the  features.  Interaction  with 

menu-driven,  using  light-pen  or  keyboard  input.  Menus  (See 
Figure  2.)  allow  various  system  choices  such  as  selection  of  the 
threshold  value,  window  size,  operating  options  and  parameters 
for  specific  applications.  Calibration,  training-by-showing, 
storing  and  loading  of  prototype  data  can  all  be  accomplished 
readi ly  . 


simply  by  showing 
performed  using  a 
a  user-selectable 
the  system  are 


The  Imaging  Sensor  -  Camera 

The  cameras  that  were  used  to  perform  the  experiments  were  GE 
TN2200  and  TN2500[3l  solid  state  camera  with  CID(Charge  Injection 
Device)  imaging  array  sensor.  For  the  latter,  the  CID  sensor 
contains  over  60000  light  sensing  picture  elements ( pi xels )  in  a 
3x4  aspect  ratio  format.  The  typical  spectral  responsivity 
function  of  CID  sensor  in  shown  in  Figure  5.2. 

Threshold  and  Histogram 

Binarization  has  been  a  central  topic  of  interest  to  computer 
vision  and  numerous  approaches  have  been  publ ished [ 15 1 [ 16 ] . 
Generally  in  binary  vision  system,  a  histogram  is  a  graph  that 
measures  the  number  of  pixels  occurring  at  grid  points  along  a 
grey  scale.  In  order  to  obtain  a  stable  image  on  the  monitor,  it 
is  necessary  to  set  the  threshold  number  at  a  point  on  the 
histogram  that  will  slice  the  grey  scale  and  redefine  the  pixels 
to  be  either  black  or  white.  The  histogram  is  a  guide  for 
accomplishing  this.  The  ideal  shape  of  the  histogram,  including 
a  properly  adjusted  camera  and  high  contrast  scene,  will  have  two 
well  defined  peaks.  One  peak  on  the  histogram  corresponds  to  the 
pixels  in  the  background,  the  other  peak  to  pixels  in  the  object. 
Setting  the  threshold  at  the  valley  that  occurs  between  the  two 
peaks  ensures  that  the  correct  pixels  will  become  either  part  of 
the  background  or  part  of  the  image.  The  image  will  then  be  a 


Figure  1.  The  MI  VS-100  binary  vision  system. 
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stable,  high  contrast  silhouette  ol  the  object  being  viewed. 


The  HISTOGRAM  command  in  the  VS-100  causes  the  system  to  sweep 
through  the  threshold  range  from  255  to  0,  displaying  the 
thresholded  image  at  each  setting  and  graphically  indicating  the 
number  of  pixels  changing  from  black  to  white  at  each  threshold 
value.  High  values  on  the  histogram  corresponds  to  threshold 
settings  where  the  image  changes  rapidly.  Low  values  correspond 
to  threshold  settings  where  the  image  is  stable.  For  most 
objects  the  optimal  setting  of  the  threshold  is  in  the  stable 
area  corresponding  to  the  low  point  between  the  two  peaks. 


Color  Image 


A  wide  range  of  colors  can  be  reproduced ,  to  the  sat 
of  the  eye,  by  the  addition  of  only  three  monochroma 
sources,  e.g.,  red,  green,  and  blue.  The  three  CIE 
primaries  are  monochromatic  light  of  wavelength  700  r 
546.1  nm  (green),  and  435.8  nm  (blue). 


action 
:  light 
andard 
v  red ) , 


A  digital  image  is  defined  by  a  function  of  2-D  posi  ,  say 
I(m,n),  defined  at  chosen  grid  points  of  the  image.  For  a 
achromatic  grey-scale  image,  the  function  I  is  scalar-valued,  its 
value  being  the  brightness  of  the  image  at  a  certain  point.  For 
color  images,  three  values  must  be  specified  at  each  point,  i.e. 
the  function  I  is  vector-valued  and  has  three  components.  A 
common  choice  of  the  three  components  is  that  of  the  so-called 
red,  green  and  blue  (R,  G  and  B)  components.  The  R,  G,  and  B 
components  can  be  transformed  to  other  quantities,  more  closely 
associated  with  our  visual  senses  of  color,  such  as  luminance, 
hue,  and  saturation. 


A  simple  and  commonly  used  transformation  is  defined  here  for 
reference . 


RC  =  R  /  (R+G+B) 

GC  =  G  /  (R+G+B) 

Y  =  aR  +  bG  +  cB 

where  a,  b,  and  c  are  suitably  chosen  scaling  constants.  Y  gives 
the  luminance  of  the  image  pixel  and  RC  and  GC  provide  the 

chromaticity  information.  Hue  and  satuaration  can  be  deduced  by 

converting  RC  and  GC  to  a  polar  coordinate  system  (Nevatia, 

1976)1 10].  It  is  important  to  notice  that  the  luminance  is 

expressed  as  the  linear  summation  of  the  contributions  from  the 
R,  G,  and  B  components.  This  formula  will  be  expressed  in  a  more 
explicit  form  in  the  later  sections. 


Luminance 


Luminance  indicates  the  amount  of  light  intensity,  which  is 


I 


1 


perceived  by  the  eye  as  brightness.  In  a  black-and-white  picture, 
the  lighter  parts  have  more  luminance  than  the  dark  areas. 
Different  colors  have  different  shades  of  luminance,  however,  as 
some  colors  appear  brighter  than  others.  The  luminance  really 
indicates  how  the  color  will  look  in  a  black-and-white 
reproduction . 

The  eye  does  not  respond  equally  to  radiated  energy  of  all 
visible  wavelengths.  There  is  wide  variation  between  observers, 
and  the  response  is  also  a  function  of  light  intensity.  Based  on 
thousands  of  measurements  on  human  observers,  the  average  eye  is 
considered  to  respond  according  to  the  luminosity  function  of  the 
standard  observer.  A  standard  luminosity  function  has  a 
Gaussian-1 ike  bell-shape  distribution  (relative  luminosity  versus 
wavelength)  which  centers  at  546  nm. 

The  luminance  of  a  surface  is  the  effect  on  the  average 
sensor(eye)  of  the  light  emitted  by  a  unit  area  of  the  surface. 
It  is  the  integrated  effect  of  the  sensor(eye)  response  y(l)  and 
the  visible  light  power  radiated  by  the  surface  E(l),  both  of 
which  are  functions  of  the  wavelength  1.  The  integration  is 
expressed  by 

Luminance  =  ^680  E(l)y(l)dl  lm/unit  area 

where  lm  is  the  abbreviation  for  lumen  and  the  radiated  power 
E  ( 1 )  is  in  watts  per  unit  area.  The  constant  680  lm/W  is  the 
luminosity  of  radiant  power  at  the  peak  of  the  luminosity  curve, 
at  546  nm. 

The  brightness  of  a  surface  is  defined  in  terms  of  a  surface 
which  reflects  the  light  in  perfectly  diffuse  fashion.  Such  a 
surface  has  a  brightness  of  1  foot-lambert  (fl)  for  each  lumen 
incident  upon  it  if  it  does  not  absorb  any  energy,  i.e.,  has 
reflectivity  of  unity. 

The  typical  CID  (charge  injection  devices)  spectral 
responsivity  curve  of  the  GE  solid  state  cameras  is  shown  in 
Figure  5.  The  curve  indicates  that  CID  also  has  the  greatest 
responsivity  to  green  color(around  560  nm).  The  responsivity 
curve  extends  far  beyond  700  nm  which  indicates  that  CID  has  a 
broader  responsive  range  than  humans  in  the  infrared  regions. 


Hue,  Saturation  and  Chrominance 

The  hue  describes  the  intrinsic  nature  of  the  color,  i.e., 
red,  green,  cyan,  purple,  etc.  The  color  itself  is  its  hue  or 
tint.  A  red  apple  has  a  red  hue;  green  leaves  have  a  green  hue. 
The  color  of  any  object  is  determined  primarily  by  its  hue. 
Different  hues  result  from  different  wavelengths  of  the  light 
producing  the  visual  sensation  in  the  eye. 

Saturation  is  a  measure  of  color  intensity,  i.e. 
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versus  vivid  quality.  Desaturated  colors  are  whitish  or  washed 
out.  Saturated  colors  are  vivid,  intense,  deep,  and  strong.  Pale 
or  weak  colors  have  little  saturation.  The  saturation  indicates 
how  little  the  colors  is  diluted  by  white.  It  is  important  to 
note  that  a  fully  saturated  color  has  no  white.  Then  the  color 
has  only  its  own  hue,  without  any  other  components  that  could  be 
added  by  the  red,  blue,  and  green  of  white. 


The  term  chrominance  is  used  to  indicate  the  combined  effect 
of  hue  and  saturation  of  a  color.  In  color  TV,  the  3.58-MHz 
modulated  subcarrier'  color  signal  more  specifically  is  the 
chrominance  signal.  The  chrominance  includes  all  the  color 
information,  without  the  brightness.  The  chrominance  and  the 
brightness  together  specify  the  picture  color  information 
completely.  Chrominance  is  also  called  "chroma". 


The  Luminance  Effect  of  Colors 


It  is  intuitively  more  understandable  to  discuss  the  luminance 
effect  of  various  colors  by  studying  the  primaries  used  in  TV 
color  signals.  The  transf ormations [ 4 ) [ 7 ]  to  obtain  the  X,  Y,  and 
Z  primaries  for  color  vision,  based  on  the  FCC  primaries,  are 
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These  X,  Y,  Z  primaries,  are  nonphysical  and  do  not  represent 
real  colors.  They  can  represent  real  electric  signals  but  must 
be  electrically  transformed  (using  an  electrical  analog  of  the 
transformation  equations)  to  R,  G,  and  B  signals  before  being 
displayed.  The  Y  signal  is  the  luminance  signal.  It  is 
representative  of  the  black-and-white  TV  signal  that  would  be 
derived  from  the  same  matter  by  a  high-performance  monochrome 
camera.  The  X  and  Z  signals  carry  the  color  information  system, 
i.e.,  the  nonluminance  content.  The  Y  (luminance  signal)  contains 
the  brightness  variations  of  the  picture  information.  This 
signal  is  formed  by  adding  the  primary  red,  green,  and  blue  video 
signals  in  the  proportions  (rounding  off  the  third  digit  below 
the  decimal  point  to  simplify  calculations)  : 

Y  =  0.30R  +  0.59G  +  0.11B 

These  percentages  correspond  to  the  relative  brightness  of  the 
three  primary  colors.  Therefore,  a  scene  reproduced  in  black  and 
white  by  the  Y  signal  looks  the  same  as  when  it  is  televised  in 
monochrome.  The  Y  signal  has  its  maximum  relative  amplitude  of 
unity,  1.0  or  100  percent  for  white,  because  it  includes  R,  G, 
and  B.  This  value  for  white  is  calculated  as 


Y  =  0.30  +  0.59  +  0.11  =  1.00 


W 


As  another  example,  the  cyan  c'  'or  includes  G  and  B  but  not  R. 
Then  the  Y  value  for  the  cyan  is  calculated  as 

Y  =0+0.59  +0.11=  0.70 

All  the  voltage  value  of  the  Y  signal  can  be  calculated  in  this 
way.  The  resulting  voltages  are  the  relative  luminance  values 
for  each  of  the  following  saturated  color.  If  the  Y  signal  alone 
were  used  to  reproduce  the  pattern,  it  would  appear  as  monochrome 
bars  shading  off  from  white  at  the  left  to  grey  in  the  center  and 
black  in  the  right. 
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colors  above  (color  encoding  technique  will  be  explained  in  more 
detailed  in  a  later  section).  All  the  colors  are  assumed  to  be 
saturated  vivid  colors.  In  natural  scenes,  however,  most  colors 
are  not  100  percent  saturated.  Then  any  color  diluted  by  white 
light  has  all  three  primaries.  Assume  70%  saturation  for  yellow. 
Now  this  color  has  two  components:  70%  saturated  yellow  and  30% 
white. 

70%  Yellow  produces  0.70R  0.70G  0.00B 

30%  White  produces  0.30R  0.30G  0.30B 

Total  output  is  1.00R  1.00G  0.30B 

Y ( luminance )  =  0.30(1.00)  +  0.59(1.00)  +  0.11(0.30)  =  0.923 

We  can  see  that  this  color  is  "brighter"  than  the  saturated 
yellow  which  has  a  luminance  value  of  0.89.  Note  that  the 
addition  of  white  to  desaturate  a  color  increases  the  luminance 
value  and  decreases  the  chrominance  value,  compared  with  100% 
saturation . 


(  I  I -D  ]  RESEARCH  RESULT.0 
ANALYTICAL  AND  EXPERIMENTAL  METHOD 


The  first  step  in  the  study  of  color  machine  vision  is  to 
extract  the  color  information  from  a  scene.  Natural  scenes  in 
general  consist  of  more  than  one  color.  Human  beings  use  color 
information  as  a  powerful  clue  in  object  recognition. 
Traditional  binary  machine  vision  has  neglected  the  color 
component  in  the  scene  by  thresholding  the  grey-scale  image.  This 
is  an  original  study  in  extracting  color  information  from  a  scene 
through  filtering,  yet  still  preserving  the  processing  speed  and 
other  advantages  of  a  "binary”  vision  system. 

The  study  of  colors  involves  usage  of  narrow-  or  broad-band 
filters  installed  in  front  of  viewing  TV  cameras  or  broad-band 
light  sources,  or  by  directing  manipulating  the  chrominance  (and 
luminance)  signals  in  a  color  TV  camera.  The  choice  of  narrow  or 
wide  band  filters  depends  on  several  factors.  The  number  of 
colors  to  be  discriminated,  the  distribution  of  these  colors  in 
the  visible  spectrum,  and  the  amount  of  processing  time  involved 
determine  how  many  filters  to  use  and  what  bandwidths  to  use. 

The  commonly  used  color  filters  are  color  additive  filters 
and  color  subtractive  filters.  Color  additive  filters  are  broad 
bandpass  and  edge  interference  filters  which  transmit  one  of  the 
three  primary  colors  (red,  green  and  blue).  Combined  in  suitable 
mixtures,  these  colors  can  produce  most  of  the  color  sensations 
which  human  vision  is  capable  of  perceiving.  The  color 
subtractive  filters  are  designed  for  use  at  45  degrees  and 
reflect  the  primary  colors  red,  green,  and  blue.  The  transmitted 
colors  are  the  complementary  colors  of  the  primaries.  These  are 
cyan  (minus  red),  magenta  (minus  green),  and  yellow  (minus  blue). 
Minus  colors  are  often  used  electronically  and  photographically 
to  reproduce  the  primary  colors.  For  example,  the  strength  of 
the  red  signal  at  a  vidicon  may  be  determined  by  obtaining  the 
white  light  signal  and  subtracting  from  it  the  minus  red  (cyan) 
signal.  In  fact,  a  minus  red  channel,  a  minus  blue  channel,  and 
a  white  light  channel  are  sufficient  to  obtain  the  information  to 
reproduce  the  primary  colors,  and  is  the  basis  of  some 
inexpensive  color  television  cameras. 

Narrow-band  color  filters  are  chosen  to  select  the  bandpass 
of  the  field  of  view.  Only  light  with  the  specified  narrow 
bandwidth  can  be  transmitted  and  thresholded.  The  advantage  of 
this  approach  is  that  color  can  be  more  accurately  acquired  and 
differentiated.  The  shortcoming  is  that  in  order  to  extract  a 
large  number  of  colors,  a  corresponding  large  number  of 
narrow-band  filters  has  to  be  utilized. 

Discrete  objects  with  distinct  single  color  regions  can  be 
differentiated  by  utilizing  the  selective  filtering  property  of 
filters.  Objects  with  several  color  patches  can  be  analyzed  in 
the  following  way.  The  individual  patches  are  analyzed  and 


features  computed.  The  identified  individual  patches  can  later  be 
"recombined"  through  utilization  of  features  such  as  area,  center 
of  blob,  pixel  counts.  The  underlying  reason  for  this  is  that 
these  particular  features  for  the  whole  object  are  the  linear 
summation  of  the  same  features  of  the  constituent  color  patches, 
assuming  that  these  multicolored  objects  are  isolated  in  the 
field  of  views  (no  overlapping  or  stacking  of  objects). 


Selecting  and  Characterizing  the  Light  Source 


Color  sensing  results  from  the  compound  spectral  effect  of 
the  color  filters,  the  light  source,  and  the  imaging  sensors. 
Manufacturers  of  filters  usually  provide  the  transmittance  curves 
for  precision  filters.  The  spectral  responsivity  functions  of 
the  sensors  and  the  spectral  power  characteristics  of  the  light 
sources  are  also  important  parameters  which  must  be  measured  or 
obtained  from  the  suppliers  of  the  instruments. 

The  human  eyes  in  general  do  not  discriminate  composite 
colors  very  well.  A  source  may  appear  white  but  has  a 
far-f rom-f lat  spectrum.  In  order  to  select  the  appropriate  light 
source,  the  authors  have  performed  a  detailed  monochromator  test 
of  all  existing  light  sources  available  in  the  laboratory.  These 
sources  include  tungsten  halogen  lamps,  fluorescent  lamps, 
infrared  and  ultraviolet  sources.  The  spectral  characteristics 
of  these  sources  are  documented  by  obtaining  the  spectrographs 
for  all  sources.  This  procedure  is  important  in  selecting  a 
light  source  which  has  relatively  flat  bandpass  in  the  visible 
spectrum. 

Decision  was  made  to  use  the  "north  sky-light"  fluorescent 
lamp  as  the  light  source  to  perform  the  experiment  to  be 
discussed.  We  chose  fluorescent  lamp  for  high  luminance  and  most 
importantly,  for  more  uniform  spectral  power  distribution  than 
other  sources  such  as  tungsten  ones(See  Figure  5.1.). 


Characterizing  the  Color  Filters 


A  similar  experiment  was  performed  to  characterize  the 
transmitting  characteristics  of  the  color  filter  sets.  The 
filters  tested  included  a  set  of  color  additive  filters,  a  set  of 
color  subtractive  filters,  and  a  set  of  color-temperature 
compensation  filters  manufactured  by  Melles  Griot  Corporation. 
The  first  set  consists  of  red,  green  and  blue  filters.  In 
addition  to  the  transmittance  curves  supplied  by  Melles  Griot, 
the  spectral  property  of  these  filters  was  actually  tested  by 
using  a  monochromator.  The  transmittance  curves  of  these  filters 
under  illumination  of  the  fluorescent  lamp  are  shown  in  Figure 
4.2  -  4.4.  These  curves  clearly  show  how  the  filters  cut  off 

certain  ranges  of  wave-lengths  in  the  spectrum. 


Figure  4.1.  The  spectral  distribution  curve  Figure  4.2.  The  transmittance  curve  for 
of  the  fluorescent  lamp  used  to  perform  the  red  filter  with  the  fluorescent  lamp  as 
the  experiments  described  in  this  paper.  the  light  source. 


Figure  4.3.  The  transmittance  curve  for  Figure  4.4.  The  transmittance  curve  for 
the  green  filter  with  the  fluorescent  lamp  the  blue  filter  with  the  fluorescent  lamp 
as  the  light  source.  as  the  light  source. 


Figure  5.1.  The  spectral  distribution  curve 
of  a  tungsten  lamp. 
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Figure  6.  The  experimental  setup  for 
color  discrimination  using  binary  processing. 


Figure  5.2.  The  typical  spectral  responsivity 
curve  of  the  C1D  solid-state  camera  used 
in  the  experiments. 


Figure  7.  Close-up  view  of  the  color 
rings  used  in  the  experiments  and  the 
filters. 
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Figure  8.1.  The  raw  video  image  of  the  Figure  8.2.  At  a  threshold  value  of  96, 
red,  green  and  blue  rings  with  the  finger  the  green  ring  appears  first  because  of  its 
pointing  at  the  blue  ring.  highest  luminance. 


Figure  8.3.  At  a  threshold  value  of  49,  Figure  8.4.  At  a  threshold  value  of  21, 
both  the'  green  and  red  ring  show  their  all  three  rings  appear  on  the  monitor, 
thresholded  images.  The  blue  ring  seems  to  be  thinner  than  the 

other  two  ring. 


Figure  9.1.  The  histogram  of  the  red  ring  Figure  9.2.  The  histogram  of  the  red  ring 
without  red  filter.  with  red  filter. 
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Figure  10.  The  classification  and  coding  technique  used  in  analyzing 
colors.  A  3-bit  code  is  associated  with  each  classified  color.  A 
maximum  of  eight  colors  can  be  classified  using  this  method. 


Rejection  of  Unwanted  Colored  Object  using  Color  Filters 

The  methods  of  discriminating  colors  using  current  machine 
vision  system  fall  into  two  categories:  methods  based  on 
selection  of  spectral  bandpass  and  methods  based  on  luminance 
effects  of  color  objects. 

The  are  two  problems  involved  in  the  latter  process.  The 
first  concerns  the  optimal  selection  of  single  threshold  for  all 
colors.  The  second  is  the  confusion  caused  by  the  fact  that 
different  colors  may  have  the  same  luminance  value.  This  makes 
any  discrimination  based  on  luminance  alone  unreliable  unless 
during  the  training  period  the  ambiguous  situations  are  avoided. 

An  optimal  solution  to  this  is  through  the  usage  of  a 
combination  of  the  above  two  methods.  Confusion  caused  by 
nonunique  mapping  between  the  color  space  and  the  luminance  space 
can  be  minimized  by  using  filters  to  eliminate  undesired  spectral 
contents.  If  a  small  number  of  filters  is  used(say,  R,  G,  and  B 
filters)  to  discriminate  colors  widely-spaced  in  the  visible 
spectrum,  then  the  filters  alone  will  be  sufficient  to 
distinguish  them  using  "binary"  color  vision.  For  colors  with 
overlapped  spectral  contents,  the  luminance  effect  of  colors  will 
be  useful  in  color-differentiation  in  combination  with  filters  in 
"binary"  vision  system. 


Luminance  Effect  of  Colored  Objects 

An  experimental  testbed  (See  Figure  6.,  7.)  was  established 
to  experiment  and  evaluate  machine  color  vision.  This  test-bed 
consisted  of  a  MI  VS-100  binary  computer  vision  system,  a  color 
filter  holder  with  adjustable  supporting  stand,  the 
color-additive  filter  set,  a  set  of  plastic  color  rings  with 
identical  shape  and  size  as  test  objects,  and  the  optional  MI 
DS-100  Machine  Vision  Development  System  (Chen  and  Milgram, 
1982 )  [ 1  ]  for  developing  special  software  for  color-coding  and 
performing  the  training  and  recognition  functions.  An  experiment 
was  performed  to  assess  the  feasibility  of  discriminating 
multiple  colors  using  single  filter.  A  set  of  color  rings  were 
prepared  as  the  testing  objects.  The  set  of  rings  used  consists 
of  red,  pink,  yellow,  and  blue.  Placed  under  a  TV  camera 
connected  to  the  MI  VS-100  vision  system,  the  blue  ring  was 
invisible  (rejected)  to  the  system  under  a  red  filter  (In  an 
automated  factory,  this  feature  allows  automatic  rejection  of 
unwanted  color  objects  in  the  background.  This  also  saves 
computing  time  by  neglecting  the  otherwise  complicating  objects 
in  connectivity  analysis).  A  comparison  of  the  filtered  and 
unfiltered  view  would  identify  the  blue  ring. 


As  explained  in  the  previous  sections,  luminance  indicates 
the  amount  of  light  intensity,  which  is  perceived  by  the  eye  as 


brightness.  Different  colors  have  different  shades  of  luminance 
as  some  colors  appear  brighter  than  the  others.  Relative 
luminosity  curves  have  been  produced  by  color  scientists  to  show 
the  hue/luminance  relationship.  The  relationship  indicates  that 
the  green  hues  between  cyan  and  orange  have  maximum  brightness. 
The  rings  showed  different  total  pixel-counts(or  area)  for 
different  colors  under  the  same  thresholding  value.  This  is  a 
"feature"  that  the  VS-100  binary  vision  system  can  utilize  to 
perform  discrimination  task  for  parts  with  the  same  feature-set 
except  differences  in  colors.  The  authors  have  successfully  used 
this  property  to  distinguish  the  colors  described. 


Figure  8. 1.-8. 4.  show  the  histograms  and  the  thresholded 
images  for  the  red,  green  and  blue  rings  at  three  different 
threshold  settings.  It  is  interesting  to  notice  that  the  green 
ring  appears  first  as  the  threshold  sweeps  from  the  bottom  to  the 
top  in  grey  levels,  followed  by  the  red  ring,  and  then  by  the 
blue  ring.  It  was  pointed  out  in  the  previous  section  that  green 
appears  brightest  in  comparison  to  the  other  undiluted  colors. 
This  phenomenon  can  also  be  easily  observed  in  the  raw  video 
image  that  shows  the  three  rings  with  a  black  background.  The 
finger  points  at  the  blue  ring  which  appears  darkest. 

Color-Coding  Technique 

Utilizing  color  filters  for  the  three  primary  colors, 
color-encoding  can  be  performed.  Basically,  this  operation  is, 
in  principle,  similar  to  the  early  design  of  color  video  cameras 
which  used  a  rotating  color-filter  wheel  triggered'  by  the 
vertical  sync  signal  for  color  separation. 

We  can  use  a  3-bit  color-encoding  technique  to  code  and 
distinguish  eight  colors.  Figure  10.  shows  the  binary  tree 
classification  method  used  in  color  encoding  and  classification. 
We  can  see  that  3-bit  encoding  works  best  for  three  primary  and 
their  complementary  saturated  colors.  For  desaturated  colors, 
because  of  the  dilution  by  white(in  other  words,  mixture  with 
equal  amount  of  red,  green,  and  blue  primaries),  ambiguity  will 
arise.  For  example,  light  green,  pink,  and  light  blue  may  all  be 
indistinguishable  from  white  in  coding  since  they  are  all  coded 
111.  Another  example  is  that  yellow  and  orange  may  both  have  the 
same  coding  110,  i.e.,  both  of  them  will  pass  the  red  and  green 
filter  test  and  fail  the  blue  filter  one,  despite  that  they  are 
different  colors  as  perceived  by  the  human  eyes. 


f II-E]  CONCLUSION 


Since  color-sensing  results  from  the  compound  spectral  effect 
of  the  light  source,  the  transmitting  medium  and  intervening 
filters,  and  the  imaging  sensors,  it  is  crucial  to  define  their 
spectral  characteristics  before  any  analytical  or  experimental 
work  be  done.  This  effort  was  described  in  the  previous 
sections.  The  importance  of  this  is  exemplified  by  the  following 
example.  Two  colors  might  look  identical  under  tungsten 
illumination (2860  K),  but  completely  different  in  daylight(6500 
K)  [191. 

Using  the  experimental  testbed,  it  was  demonstrated  that:  (1) 
Under  suitable  thresholding  and  filtering,  the  "spectral  content" 
of  the  color  scenes  can  be  artificially  controlled.  (2)  Unwanted 
color  objects  can  be  easily  rejected.  (3)  Under  single  threshold 
selection,  discrimination  of  objects  identical  in  shapes  and 
sizes  but  different  in  colors  can  be  performed  using  features 
such  as  area,  pixel  counts,  perimeter,  etc.  This  is  due  to  the 
luminance  effect  of  different  colors  which  manifest  themselves  in 
thresholded  views  as  size  shrinkage  or  expansion.  (4)  Using  R-G-B 
filtering,  3-bit  color-encoding  can  be  performed  for  8  colors. 
If  we  assign  two  bits  for  each  color  and  expand  from  3-bit  to 
6-bit,  64  colors  can  be  encoded. 

The  unique  aspect  of  this  research  is  that  color  is 
acquired  together  with  all  other  features  in  a  "binary"  machine 
vision  system.  Traditional  color  discrimination  devices  such  as 
monochromators  or  spectrophotometers  analyze  only  a  beam  of  light 
instead  of  a  scene  viewed  through  a  camera  as  perceived  by  the 
human  eyes.  By  combining  MI  VS-100  machine  vision  system  with 
color  filters,  color,  as  well  as  location,  shape  and  orientation 
descriptors  can  be  easily  obtained.  This  combined 
color/location/orientation  information  can  be  used  by  a  robot  arm 
or  other  manipulators  to  distinguish  among  colored  objects  for 
part  pickup.  Additionally,  through  usage  of  filters,  irrelevant 
colors  can  be  rejected.  This  is  particularly  important  in  saving 
computer  processing  time  by  filtering  away  background  clutter. 
Objects  with  connected  multicolor  regions  can  be  recombined  by 
using  other  features  such  as  area,  center,  and  pixel  counts. 

One  form  of  logical  image  combination  arises  when  a  single 
object  consists  of  adjacent  regions  of  different  colors.  In  this 
case,  no  single  color  can  be  expected  to  segment  the  object 
entirely.  However,  it  is  possible  to  piece  together  the  object 
as  the  union  of  adjacent  (or  nonadjacent)  regions.  Generally, 
the  shape  features  computed  by  the  MI  VS-100  system  are  such  that 
the  features  for  the  union  of  non-overlapping  regions  can  be 
derived  from  the  constituent  regions'  features.  Thus  it  is 
possible  to  extract  multi-color  object  from  the  scene. 
Segmenting  regions  in  the  color  domain,  color  codes  can  be 
identified  on  a  complex  background.  In  addition,  the  limitation 
posed  by  connectivity  analysis  can  be  removed.  For  example,  a  red 
object  in  contact  with  a  blue  one  can  be  perceived  as  isolated 


ones  through  filtered  views.  The  same  scene  will  be  easily 
misinterpreted  by  conventional  binary  vision  system  as  single 
object  because  of  the  usage  of  blob  processing  and  run-length 
encoding . 

It  is  easy  to  imagine  that  the  introduction  of  color 
differentiations  for  current  machine  vision  processing  will 
greatly  extend  the  range  of  applications.  Additional  research  is 
needed  to  demonstrate  the  advantages  of  coupling  color  filtering 
and  binary  processing  so  as  to  achieve  a  powerful  tool  for 
industrial  computer  vision  applications. 
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ABSTRACT 

This  paper  iddresses  the  questions  of  the  near-term  utilization  of  color  by  existing 
state-of-the-art  binary  vision  systems.  The  introduction  of  color  discriminations  for 
current  machine  vision  processing  can  significantly  extend  the  range  of  applications.  We 
describe  our  initial  attempts  at  the  recognition  of  binary  regions  derived  from  color 
filtered  views. 

INTRODUCTION 


Color -perception  is  without  doubt  one  of  our  most  marvelous  achievements,  and  it 
exerts  profound  influences  on  industry,  business,  science,  engineering,  as  well  as  art  and 
design.  It  is  also  one  of  the  strongest  visual  cues  (others  are  edges,  grey-scale,  depth 
and  shades,  etc.)  that  we  use  to  distinguish  far/near  objects,  and  probably  the  only  cue 
to  distinguish  colored  objects  with  same  shape,  size,  and  features.  Color -perception  can 
be  considered  as  a  very  personal  experience.  It  is  affected  by  the  physical 
characteristics  of  the  source  and  the  transmitting  medium,  by  the  physiology  of  the  retina 
and  the  visual  nervous  system,  and  also  by  the  psychological  state  of  processes  of  the 
cerebral  cortex  in  interpreting  the  signals  sent  through  the  retinocortical  pathway.  The 
retinex*  (retina-and-cortex)  theory  of  human  color  vision  basically  treats  a  color  as  a 
three-part  report  from  the  retina,  independent  of  the  flux  of  radiant  energy.  The  human 
eye  has  evolved  to  see  the  world  in  unchanging  colors,  regardless  of  always  shifting, 
unpredictable  and  uneven  illumination.  This  "insensitivity"  to  changing  lighting  makes  it 
attractive  (or  industrial  applications. 

Color  vision  is  particularly  useful  in  industrial  automation  to  recognize,  search,  sort, 
and  manipulate  colored  parts  or  color-coded  objects.  The  use  of  color  permits  part 
discrimination  where  grey-scale  information  alone  is  insufficient  and  often  times  avoids  the 
complicated  and  time-consuming  grey-scale  analysis  by  using  thresholding.  The  purpose  of 
this  research  is  to  demonstrate  that  color,  together  with  position,  size,  and  shape 
informations  can  be  extracted  from  a  scene  viewed  through  a  TV  camera.  Commercial 
(black  and  white)  binary  vision  systems  have  begun  to  substantially  affect  productivity  in 
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manufacturing.  Their  value  results  from  their  programmed  ability  to  locate,  measure,  and 
identify  significant  visual  characteristics  of  manufactured  objects  or  processes.  At  present, 
these  systems  work  on  data  which  have  been  binarized  through  the  careful  control  of 
threshold  and  lighting  in  order  to  provide  high-contrast  images.  This  has  given  rise  to 
an  image  processing  technology  based  on  the  extraction  of  silhouettes  of  workpieces. 
Such  silhouettes  are  sufficient  to  accomplish  inspection,  recognition,  and  mensuration  and 
to  determine  part  position  and  orientation. 

The  study  of  machine  binary  color  vision  must  deal  with  acquisition  of  colors, 
differentiation  of  colors,  filtering  and  selective  processing  of  colors,  measurements  of 
colors  (colorimetry),  and  together  with  all  the  features  (such  as  size,  shape,  and  position 
etc.)  that  a  binary  vision  system  can  extract  from  a  scene. 

In  the  past,  the  reports  of  machine  color  vision  for  industrial  applications  have  been 
sparse.  One  example  was  the  research  done  by  I to*’V  197 5).  He  utilized  color  information 
in  an  inspection  system  for  1C  mask  patterns.  Patterns  of  1C  masks  were  illuminated  by 
red,  green,  and  blue  light,  respectively.  Defects  of  these  mask  patterns  could  be 
identified  by  the  optical  compositions  of  the  patterns  under  illumination  of  the  primary 
color  light  sources.  Loughlin  M982)*  reported  lnspectrum  -  a  full  color  inspection  system 
which  checks  the  shape  of  an  industrial  part  and  provides  detailed  qualitative  inspection 
of  surface  color.  Basically,  this  system  permits  part  discrimination  through  decoding  the 
PAL  video  signal,  digitizing  the  separated  R,  C,  and  B  components  and  storing  the  data 
in  buffer  memory  for  later  comparison.  Position  information  can  be  obtained  through  the 
"inspection  points"  under  joystick  software  control.  One  disadvantage  of  this  approach  is 
that  position  information  is  not  acquired  automatically  but  requires  human  interaction. 
Fiorini 1  (1982)  reported  a  color  sensing  system  which  functions  as  an  intelligent 
peripheral  unit  with  a  supervisory  computer  which  can  program  up  to  256  saturation  levels 
for  each  chromatic  components:  red,  green,  and  blue.  This  peripheral  unit  sends  back 
three  bits,  one  for  each  component,  flagging  overflow  or  underflow  of  programmed 
saturation  levels  by  the  chromatic  contents  of  the  sensed  object.  This  system  does  not 
provides  the  size,  shape,  and  position  information  about  the  inspected  parts.  It  is 
capable  of  sensing  moving  objects  and  is  insensitive  to  lighting  change  in  the 
environment. 


VS-100  System 

The  VS-100  vision  system  (See  Figure  I.)  is  a  commercial  binary  vision  system.  It 
receives  a  video  grey-scale  image  from  a  solid-state  or  vidicon  camera  and  thresholds  it 
into  a  binary(black/white)  image  that  is  run-length  encoded  for  data  compression  and 
subsequent  processing.  Computer  algorithms  perform  a  connectivity  analysis  of  the 
encoded  images,  building  data  structures  that  represent  essential  features  of  each 
contiguous  region.  The  vision  system  characterizes  blobs  on  the  basis  of  distinguishing 
features  such  as  area,  perimeter,  minimum  and  maximum  radii,  and  number  of  holes  (See 
Figure  3.).  The  system  can  be  trained  to  analyze  new  objects  simply  by  showing  them  to 
the  system.  Object  recognition  is  performed  using  a  nearest-neighborhood  classifier 

operating  on  a  user -selectable  set  of  the  features.  Interaction  with  the  system  are 
menu-driven,  using  light-pen  or  keyboard  input.  Menus  (See  Figure  2.)  allow  various 
system  choices  such  as  selection  of  the  threshold  value,  window  size,  operating  options 
and  parameters  for  specific  applications.  Calibration,  training-by-showing,  storing  and 
loading  of  prototype  data  can  ail  be  accomplished  readily. 


The  Imaging  Sensor  -  Camera 

The  cameras  that  were  used  to  perform  the  experiments  were  GE  TN2200  and 
TN25005  solid  state  camera  with  ClD(charge  Injection  Device)  imaging  array  sensor.  For 
the  latter,  the  CID  sensor  contains  over  60000  light  sensing  picture  elements(pixels)  in  a 
3x9  aspect  ratio  format.  The  typical  spectral  responsivity  function  of  CID  sensor  in 
shown  in  Figure  5.2. 
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Figure  2.  A  typical  menu  of  the  VS-100  Figure  3.  Display  of  feature  names  and 
system,  values. 


BACKGROUND  of  binary  and  color  vision 


Threshold  and  Histogram 

In  binary  vision  system,  a  histogram  is  a  graph  that  measures  the  number  of  pixels 

occurring  at  grid  points  along  a  grey  scale.  In  order  to  obtain  a  stable  image  on  the 
monitor,  it  is  necessary  to  set  the  threshold  number  at  a  point  on  the  histogram  that 
will  slice  the  grey  scale  and  redefine  the  pixels  to  be  either  black  or  white.  The 

histogram  is  a  guide  for  accomplishing  this.  The  ideal  shape  of  the  histogram,  including 
a  properly  adjusted  camera  and  high  contrast  scene,  will  have  two  well  defined  peaks. 
One  peak  on  the  histogram  corresponds  to  the  pixels  in  the  background,  the  other  peak 
to  pixels  in  the  object.  Setting  the  horizontal  arrow  at  the  valley  that  occurs  between 
the  two  peaks  ensures  that  the  correct  pixels  will  become  either  part  of  the  background 
or  part  of  the  image.  The  image  will  then  be  a  stable,  high  contrast  silhoulette  of  the 
object  being  viewed. 

The  HISTOGRAM  command  in  the  VS-100  causes  the  system  to  sweep  through  the 
threshold  range  from  255  to  0,  displaying  the  thresholded  image  at  each  setting  and 

graphically  indicating  the  number  of  pixels  changing  from  black  to  white  at  each 
threshold  value.  High  values  on  the  histogram  corresponds  to  threshold  settings  where 
the  image  changes  rapidly.  Low  values  correspond  to  threshold  settings  where  the  image 
is  stable.  For  most  objects  the  optimal  setting  of  the  threshold  is  in  the  stable  area 

corresponding  to  the  low  point  between  the  two  peaks. 

Color  Image 

A  wide  range  of  colors  can  be  reproduced,  to  the  satisfaction  of  the  eye,  by  the 
addition  of  only  three  monochromatic  light  sources,  e.g.,  red,  green,  and  blue.  The  three 
C1E  standard  primaries  are  monochromatic  light  of  wavelength  700  nm  (red),  546.1  nm 
(green),  and  435.8  nm  (blue). 

A  digital  image  is  defined  by  a  function  of  2-D  position,  say  llm,n),  defined  at 
chosen  grid  points  of  the  image.  For  a  achromatic  grey-scale  image,  the  function  I  is 
scalar -valued,  its  value  being  the  brightness  of  the  image  at  a  certain  point.  For  color 
images,  three  values  must  be  specified  at  each  point,  i.e.  the  function  I  is  vector -valued 
and  has  three  components.'  A  common  choice  of  the  three  components  is  that  of  the 

so-called  red,  green  and  blue  (R,  G  and  B)  components.  The  R,  G,  and  13  components  can 

be  transformed  to  other  quantities,  more  closely  associated  with  our  visual  senses  of 
color,  such  as  brightness,  hue,  and  saturation.' 

A  simple  and  commonly  used  transformation  is  defined  here  for  reference. 

RC  =  R  /  (R*G*13) 

GC  =  G  /  (R.G.15) 

Y  =  aR  ♦  bG  ♦  cB 

where  a,  b,  and  c  are  suitably  chosen  constants.'  Y  gives  the  luminance  of  the  image 

pixel  and  RC  and  GC  provide  the  chromaticity  information.  Hue  and  satuaration  can  be 
deducted  by  converting  RC  and  GC  to  a  polar  coordinate  system  (Nevatia,  1976).10  It  is 

important  to  notice  that  the  luminance  is  expressed  as  the  linear  summation  of  the 
contributions  from  the  R,  G,  and  13  components.  Th.s  formula  will  be  expressed  in  a 
more  explicit  form  in  the  later  sections. 


Luminance 


Luminance  indicates  the  amount  of  light  intensity,  which  is  perceived  by  the  eye  as 
brightness.  In  a  black-and-white  picture,  the  lighter  parts  have  more  luminance  than  the 
dark  areas.  Different  colors  have  different  shades  of  luminance,  however,  as  some  colors 
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appear  brighter  than  others.  The  luminance  really  indicates  how  the  color  will  look  in  a 
black-and-white  reproduction. 


The  eye  does  not  respond  equally  to  radiated  energy  of  all  visible  wavelengths. 

There  is  wide  variation  between  observers,  and  the  response  is  also  a  function  of  light 
intensity.  Based  on  thousands  of  measurements  on  human  observers,  the  average  eye  is 
considered  to  respond  according  to  the  luminosity  function  of  the  standard  observer.  A 

standard  luminosity  function  has  a  Caussian-like  bell-shape  distribution  (relative  luminosity 

versus  wavelength)  which  centers  at  546  nm. 

The  luminance  of  a  surface  is  the  effect  on  the  average  sensor(eye)  of  the  light 
emitted  by  a  unit  area  of  the  surface.  It  is  the  integrated  effect  of  the  sensor(eye) 

response  y(/)  and  the  visible  light  power  radiated  by  the  surface  E(l),  both  of  which  are 
functions  of  the  wavelength  I.  The  integration  is  expressed  by 

Luminance  -J  6S0  E(|)y(l)dl  lm/unit  area 

where  Im  is  the  abbreviation  for  lumen  and  the  radiated  power  E(l)  is  in  watts  per  unit 
area.  The  constant  680  lm/W  is  the  luminosity  of  radiant  power  at  the  peak  of  the 
luminosity  curve,  at  546  nm. 

The  brightness  of  a  surface  is  defined  in  terms  of  a  surface  which  reflects  the  light 
in  perfactly  diffuse  fashion.  Such  a  surface  has  a  brightness  of  1  foot-lambert  (fl)  for 
each  lumen  incident  upon  it  if  it  does  not  absorb  any  energy,  i.e.,  has  reflectivity  of 
unity. 

The  typical  C1D  (charge  injection  deveices)  spectral  responsivity  curve  of  the  GE 
solid  state  cameras  is  shown  in  Figure  5.  The  curve  indicates  that  C1D  also  has  the 
greatest  responsivity  to  green  color(around  560  nm).  The  responsivity  curve  extends  far 
beyond  700  nm  which  indicates  that  CID  has  a  broader  responsive  range  than  humans  in 
the  infrared  regions. 


Hue,  Saturation  and  Chrominance 

The  hue  describes  the  intrinsic  nature  of  the  color,  i.e.,  red,  green,  cyan,  purple, 

etc.  The  color  itself  is  its  hue  or  tint.  A  red  apple  has  a  red  hue;  green  leaves  have  a 

green  hue.  The  color  of  any  object  is  determined  primarily  by  its  hue.  Different  hues 
result  from  different  wavelengths  of  the  light  producing  the  visual  sensation  in  the  eye. 

Saturation  is  a  measure  of  color  intensity,  i.e.  its  pastel  versus  vivid  quality. 
Desaturatcd  colors  are  whitish  or  washout.  Saturated  colors  are  vivid,  intense,  deep,  and 
strong.  Pale  or  weak  colors  have  little  saturation.  The  saturation  indicates  how  little  the 
colors  is  diluted  by  white.  It  is  important  to  note  that  a  fully  saturated  color  has  no 
white.  Then  the  color  has  only  its  own  hue,  without  any  other  components  that  could  be 

added  by  the  red,  blue,  and  green  of  white. 

The  term  chrominance  is  used  to  indicate  both  hue  and  saturation  of  a  color.  In 
color  TV,  the  3.58-MHz  color  signal  more  specifically  is  the  chrominance  signal.  The 
chrominance  includes  all  the  color  information,  without  the  brightness.  The  chrominance 
and  the  brightness  together  specify  the  picture  color  information  completely. 
Chrominance  is  also  called  "chroma". 


The  Luminance  Effect  of  Colors 


It  is  intuitively  more  understandable  to  discuss  the  luminance  effect  of  various  colors 
by  studying  the  primaries  used  in  TV  color  signals.  The  transformations  to  obtain  the  X, 
Y,  and  2  primaries  for  color  vision,  based  on  the  FCC  primaries,  are4  7 
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X 

= 

0.608  0.176  0.200 

U 

Y 

- 

0.299  0.587  0.1  14 

c 

7 

= 

0.000  0.066  1.112 

B 

\  ( 

These  X,  Y,  Z  primaries,  are  nonphysical  and  do  not  represent  real  colors.  They  can 
represent  real  electric  signals  but  must  be  electrically  transformed  (using  an  electrical 
analog  of  the  transformation  equations)  to  R,  C,  and  B  signals  before  being  displayed. 
The  Y  signal  is  the  luminance  signal.  It  is  representative  of  the  black-and-white  TV 
signal  that  would  be  derived  from  the  same  matter  by  a  high-performance  monochrome 
camera.  The  X  and  Z  signals  carry  the  color  information  system,  i.e.,  the  nonluminance 
content.  The  Y  (luminance  signal)  contains  the  brightness  variations  of  the  picture 
information.  This  signal  is  formed  by  adding  the  primary  red,  green,  and  blue  video 
signals  in  the  proportions  (rounding  off  the  third  digit  below  the  decimal  point  to  simplify 
calculations)  : 

Y  =  0.30R  .  0.59G  .  0.1  IB 

These  percentages  correspond  to  the  relative  brightness  of  the  three  primary  colors. 
Therefore,  a  scene  reproduced  in  black  and  white  by  the  Y  signal  looks  the  same  as 

when  it  is  televised  in  monochrome.  The  Y  signal  has  its  maximum  relative  amplitude  of 
unity,  1.0  or  100  percent  for  white,  because  it  includes  R,  G,  and  B.  This  value  for 

white  is  calculated  as 

Y  =  0.30  .  0.59  *  0.11  =  1.00 

As  another  example,  the  cyan  color  includes  G  and  B  but  not  R.  Then  the  Y  value  for 
the  cyan  is  calculated  as 

Y  i  0  ♦  0.59  .0.11  =  0.70 

,  All  the  voltage  value  of  the  Y  signal  can  be  calculated  in  this  way.  The  resulting 

voltages  are  the  relative  luminance  values  for  each  of  the  following  saturated  color.  If 
the  Y  signal  alone  were  used  to  reproduce  the  pattern,  it  would  appear  as  monochrome 

bars  shading  off  from  white  at  the  left  to  grey  in  the  center  and  black  in  the  right. 


white 

yellow 

cyan 

green 

magenta  red 

blue 

black 

1.00 

0.89 

0.70 

0.59 

0.41 

0.30 

0.1 1 

0.00 

111 

110 

010 

101 

100 

100 

001 

000 

The 

second 

line  indicates  the 

binary 

coding  for 

the  eight 

colors  above  (color 

encoding  technique  will  be  explained  in  more  detailed  in  a  later  section).  All  the  colors 
are  assumed  to  be  saturated  vivid  colors.  In  natural  scenes,  however,  most  colors  are 
not  100  percent  saturated.  Then  any  color  diluted  by  white  light  has  all  three 


primaries.'  Assume  70%  saturation  lor 

yellow. 

Now  this  color  has  two  components:  70% 

saturated  yellow  and  30% 

white. 

70%  Yellow  produces 

0.70R 

0.70G 

0.00B 

30%  White  produces 

0.30R 

0.30G 

0.30B 

Total  output  isl.OOR 

1.00G 

0.30B 

Y(luminance)  =  0.30(1.00)  ♦  0.59(1.00)  .  0.11(0.30)  =  0.923 


We  can  see  that  this  color  is  "brighter"  than  the  saturated  yellow  which  has  a  luminance 
value  of  0.89.  Note  that  the  addition  of  white  to  desaturate  a  color  increases  the 
luminance  value  and  decreases  the  chrominance  value,  compared  with  100%  saturation. 
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ANALYTICAL  AND  EXPERIMENTAL  METHOD 

The  first  step  in  the  study  of  color  machine  vision  is  to  extract  the  color 
information  from  a  scene.  Natural  scenes  in  general  consist  of  more  than  one  color. 
Human  beings  use  color  information  as  a  powerful  clue  in  object  recognition.  Traditional 
binary  machine  vision  has  neglected  the  color  component  in  the  scene  by  thresholding  the 
grey-scale  image.  This  is  an  original  study  in  extracting  color  information  from  a  scene 
through  filtering,  yet  still  preserving  the  processing  speed  and  other  advantages  of  a 

"binary"  vision  system. 

The  study  of  colors  involves' usage  of  narrow-  or  broad-band  filters  installed  in  front 
of  viewing  TV  cameras  or  broad-band  light  sources,  or  by  directing  manipulating  the 
chrominance  (and  luminance)  signals  in  a  color  TV  camera.  The  choice  of  narrow  or  wide 
band  filters  depends  on  several  factors.  The  number  of  colors  to  be  discriminated,  the 
distribution  of  these  colors  in  the  visible  spectrum,  and  the  amount  of  processing  time 

involved  determine  how  many  filters  to  use  and  what  bandwidths  to  use. 

The  commonly  used  color  filters  are  color  additive  filters  and  color  subtractive 
filters.  Color  additive  filters  used  are  broad  bandpass  and  edge  interference  filters  which 
transmit  one  of  the  three  primary  colors  (red,  green  and  blue).  Combined  in  suitable 
mixtures,  these  colors  can  produce  most  of  the  color  sensations  which  human  vision  is 
capable  of  perceiving.  The  color  subtractive  fitters  are  designed  for  use  at  45  degrees 
and  reflect  the  primary  colors  red,  green,  and  blue.  The  transmitted  colors  are  the 
complementary  colors  of  the  primaries.  These  are  cyan  (minus  red),  magenta  (minus 

green),  and  yellow  (minus  blue).  Minus  colors  are  often  used  electronically  and 

photographically  to  reproduce  the  primary  colors.  For  example,  the  strength  of  the  red 
signal  at  a  vidicon  may  be  determined  by  obtaining  the  white  light  signal  and  subtracting 
from  it  the  minus  red  (cyan)  signal.  In  fact,  a  minus  red  channel,  a  minus  blue  channel, 
and  a  white  light  channel  are  sufficient  to  obtain  the  information  to  reproduce  the 
primary  colors,  and  is  the  basis  of  some  inexpensive  color  television  cameras. 

Narrow-band  color  filters  arc  chosen  to  select  the  bandpass  of  the  field  of  view. 

Only  light  with  the  specified  narrow  bandwidth  can  be  tramsmitted  and  thresholded.  The 
advantage  of  this  approach  is  that  color  can  be  more  accurately  acquired  and 

differentiated.  The  shortcoming  is  that  in  order  to  process  a  large  number  of  colors,  a 

corresponding  large  number  of  narrow-band  filters  has  to  be  utilized. 

Discrete  objects  with  distinct  single  color  regions  can  be  easily  differentiated. 
Objects  with  several  color  patches  can  be  analyzed  in  the  following  way.  The  individual 

patches  are  analyzed  and  features  computed.  The  identified  individual  patches  can  later 
be  "recombined"  through  utilization  of  features  such  as  area,  center  of  blob,  pixel 
counts. 


Selecting  and  Characterizing  the  Light  Source 

Color  sensing  results  from  the  compound  spectral  effect  of  the  color  filters,  the 
light  source,  and  the  imaging  sensors.  Manufacturers  of  filters  usually  provide  the 
transmittance  curves  for  precision  filters.  The  spectral  responsivity  functions  of  the 

sensors  and  the  spectral  power  characteristics  of  the  light  sources  are  also  important 

parameters  which  must  be  measured  or  obtained  from  the  suppliers  of  the  instruments. 

The  human  eyes  in  general  do  not  discriminate  composite  colors  very  well.  A  source 
may  appear  white  but  has  a  far-from-flat  spectrum.  In  order  to  select  the  appropriate 
light  source,  the  authors  have  performed  a  detailed  monochromator  test  of  all  existing 
light  sources  available  in  the  laboratory.  These  sources  include  tungsten  halogen  lamps, 

fluorescent  lamps,  infrared  and  ultraviolet  sources.  The  spectra)  characteristics  of  these 
sources  are  documented  by  obtaining  the  spectrographs  for  all  sources.  This  procedure  is 
important  in  selecting  a  light  source  which  has  relatively  flat  bandpass  in  the  visible 

spectrum. 
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Decision  was  made  to  use  the  "north  sky-light"  fluorescent  lamp  as  the  light  source 
to  perform  the  experiment  to  be  discussed.  We  chose  fluorescent  lamp  for  high 
luminance  and  most  importantly,  for  more  uniform  spectral  power  distribution  than  other 
sources  such  as  tungsten  onesfSee  Figure  5.J.). 


Characterizing  the  Color  Filters 


A  similar  experiment  was  performed  to  characterize  the  transmitting  characteristics 
of  the  color  filter  sets.  The  filters  tested  included  a  set  of  color  additive  filters,  a  set 
of  color  subtractive  filters,  add  a  set  of  color-temperature  compensation  filters 
manufactured  by  .Melles  Criot  Corporation.  The  first  set  consists  of  red,  green  and  blue 
filters.  In  addition  to  the  transmittance  curves  supplied  by  Melles  Griot,  the  spectral 
property  of  these  filters  was  actually  tested  by  using  a  monochromator.  The 
transmittance  curves  of  these  filters  under  illumination  of  the  fluorescent  lamp  are  shown 
in  Figure  4.2  -  4.4.  These  curves  clealy  show  how  the  filters  cut  off  certain  ranges  of 
wave-lengths  in  the  spectrum. 


Rejection  of  Unwanted  Colored  Object  using  Color  Filters 

The  methods  of  discriminating  colors  using  current  machine  vision  system  fall  into 

two  categories:  methods  based  on  slection  of  spectral  passband  and  methods  based  on 
luminance  effects  of  color  objects. 

The  are  two  problems  involved  in  the  latter  process.  The  first  concerns  the  optimal 

selection  of  single  threshold  for  all  colors.  The  second  is  the  confusion  caused  by  the 
(act  that  different  colors  may  have  the  same  luminance  value.  This  makes  any 
discrimination  based  on  luminance  alone  unreliable  unless  during  the  training  period  the 
ambiguous  situations  are  avoided. 

An  optimal  solution  to  this  is  through  the  usage  of  a  combination  of  the  above  two 
methods.  Confusion  caused  by  nonunique  mapping  between  the  color  space  and  the 

luminance  space  can  be  minimized  by  using  filters  to  eliminate  undesired  spectral 
contents.  If  a  small  number  of  filters  is  usedfsay,  R,  G,  and  0  filters)  to  discriminate 
colors  widely-spaced  in  the  visible  spectrum,  then  the  filters  alone  will  be  sufficient  to 
distinguish  them  using  "binary"  color  vision.  For  colors  with  overlapped  spectral  contents, 

the  luminance  effect  of  colors  will  be  useful  in  color-differentiation  in  combination  with 

filters  in  "binary"  vision  system. 


Luminance  Uffect  of  Colored  Pbiects 


An  experimental  testbed  (5ee  Figure  6.,  7.)  was  established  to  experiment  and 

evaluate  machine  color  vision.  This  test-bed  consisted  of  a  Ml  VS-100  binary  computer 

vision  system,  a  color  filter  holder  with  adjustable  supporting  stand,  the  color -additive 

filter  set,  a  set  of  plastic  color  rings  with  identical  shape  and  size  as  test  objects,  and 

the  optional  Ml  DS-100  Machine  Vision  Development  System'fChen  and  Milgram,  1982)  for 
developing  special  software  for  color-coding  and  performing  the  training  and  recognition 

functions.  An  experiment  was  performed  to  assess  the  feasibility  of  discriminating  multiple 
colors  using  single  filter.  A  set  of  color  rings  were  ,  -epared  as  the  testing  objects. 

The  set  of  rings  used  consists  of  red,  pin  .  yellow,  and  blue.  Placed  under  a  TV  camera 
connected  to  the  Ml  VS-100  vision  system,  the  blue  ring  was  invisible  (rejected)  to  the 

system  under  a  red  filter  (In  an  automated  factory,  tin.,  feature  allows  automatic 

reaction  of  unwanted  color  objects  in  the  background.  This  also  saves  computing  time  by 

neglecting  the  otherwise  complicating  objects  in  connectivity  analys.s).  A  comparison  of 
the  filtered  and  unfiltcred  view  would  identify  the  blue  ring. 
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wavelengths( nm) 

(starting  at  350  nm,  with  65nm  spacing) 

Figure  4.1.  The  spectral  distribution  curve  Figure  4.2.  The  transmittance  curve  for 
of  the  fluorescent  lamp  used  to  perform  the  red  filter  with  the  fluorescent  lamp  as 
the  experiments  described  in  this  paper.  the  light  source. 


Figure  4.3.  The  transmittance  curve  for  Figure  4.4.  The  transmittance  curve  for 
the  green  filter  with  the  fluorescent  lamp  the  blue  filter  with  the  fluorescent  lamp 
as  the  light  source.  as  the  light  source. 
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Figure  5.1.  The  spectral  distribution  curve  Figure  5.2.  The  typical  spectral  responsivity 
of  a  tungsten  lamp.  curve  of  the  CID  solid-state  camera  used 

in  the  experiments. 
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Figure  6.  The  experimental  setup  for  Figure  7.  Close-up  view  of  the  color 
color  discrimination  using  binary  processing.  rings  used  in  the  experiments  and  the 

filters. 


Figure  8.1.  The  raw  video  image  of  the  Figure  8.2.  At  a  threshold  value  of  96, 
red,  green  and  blue  rings  with  the  finger  the  green  ring  appears  first  because  of  its 
pointing  at  the  blue  ring.  highest  luminance. 


Figure  8.5.  At  a  threshold  value  of  49,  Figure  8.4.  At  a  threshold  value  of  21, 
both  the  green  and  red  ring  show  their  all  three  rings  appear  on  the  monitor, 
threshoided  images.  The  blue  ring  seems  to  be  thinner  than  the 

other  two  ring. 


It 


Figure  9.1.  The  histogram  of  the  red  ring 
without  red  filter. 


Figure  9.2.  The  histogram  of  the  red  ring 
with  red  filter. 


Red 


Figure  10.  The  classification  and  coding  technique  used  in  analyzing 
colors.  A  3-bit  code  is  associated  with  each  classified  color.  A 
maximum  of  eight  colors  can  be  classified  using  this  method. 
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As  explained  in  the  previous  sections,  luminance  indicates  the  amount  ol  light 
intensity,  which  is  perceived  by  the  eye  as  brightness.  Different  colors  have  different 
shades  of  luminance  as  some  colors  appear  brighter  than  the  others.  Relative  luminosit 
curves  have  been  produced  by  color  scientists  to  show  the  hue/luminance  relationship. 
The  relationship  indicates  that  the  green  hues  between  cyan  and  orange  have  maximum 
brightness.  The  rings  showed  different  total  pixel-countsior  area)  for  different  colors 
under  the  same  thresholding  value.  This  is  a  "feature”  that  the  VS-100  binary  vision 
system  can  utilize  to  perform  discrimination  task  for  parts  with  the  same  feature-set 
except  differences  in  colors.  The  authors  have  successfully  used  this  property  to 
distinguish  the  colors  described. 

Figure  8.1. -8.4.  show  the  histograms  and  the  thresholded  images  for  the  red,  green 
and  blue  rings  at  three  different  threshold  settings.  It  is  interesting  to  notice  that  the 
green  ring  appears  first  as  the  threshold  sweeps  from  the  bottom  to  the  top  in  grey 
levels,  followed  by  the  red  ring,  and  then  by  the  blue  ring.  It  was  pointed  out  in  the 
orevious  section  that  green  appears  brightest  in  comparison  to  the  other  undiluted  colors. 
This  phenomenon  can  also  be  easily  observed  in  the  raw  video  image  that  shows  the  three 
rings  with  a  black  background.  The  finger  points  at  the  blue  ring  which  appears  darkest. 

Color-Coding  Technique 

Utilizing  color  filters  for  the  three  primary  colors,  color -encoding  can  be  performed. 
Basically,  this  operation  is,  in  principle,  similar  to  the  early  design  of  color  video 

cameras  which  used  a  rotating  color-filter  wheel  triggered  by  the  vertical  sync  signal  for 
color  separation. 

Me  can  use  a  3-bit  color -encoding  technique  to  code  and  distinguish  eight  colors. 
Figure  10.  shows  the  binary  tree  classification  method  used  in  color  encoding  and 
classification.  M'e  can  see  that  3-bit  encoding  works  best  for  three  primary  and  their 
complementary  saturated  colors.  For  desaturated  colors,  because  of  the  dilution  by 
wh.tefin  other  words,  mixture  with  equal  amount  of  red,  green,  and  blue  primaries), 
ambiguity  wiil  arise.  For  example,  light  green,  pink,  and  light  blue  may  all  be 

indistinguishable  from  white  in  coding  since  they  are  all  coded  00 0.  Another  example  is 
that  yellow  and  orange  may  both  have  the  same  coding  110,  i.fe.,  both  of  them  will  pass 
the  red  and  green  filter  test  and  fail  the  blue  filter  one,  despite  that  they  are  different 
colors  as  perceived  by  the  human  eyes. 


CONCLUSION 

The  unique  aspect  about  this  research  is  that  color  is  acquired  together  with  all 
other  features  in  a  "binary"  machine  vision  system.  Traditional  color  discrimination 
devices  such  as  monochromators  or  spectrophotometers  only  analyze  a  beam  of  light 

instead  of  a  scene  viewed  through  a  camera  as  perceived  by  the  human  eyes.  By 

combining  Ml  VS-100  machine  vision  system  with  color  filters,  color,  as  well  as  location, 

shape  and  orientation  descriptors  can  be  easily  obtained.  This  combined 

color/location/orientation  information  can  be  used  by  a  robot  arm  or  other  manipulators  to 
pick  up  a  distinguished  colored  object.  Additionally,  through  usage  of  filters,  irrelevant 
colors  can  be  easily  rejected.  This  is  particularly  important  in  saving  computer  processing 

time  by  filtering  away  cluttered  background  noise.  Objects  with  connected  multicolor 
regions  can  be  recombined  by  using  other  features  such  as  area,  center,  and  pixel 
counts.  Color  codes  can  be  identified  on  a  complex  background.  The  limitation  posed  by 
connectivity  analysis  can  be  easily  removed.  For  example,  a  red  object  in  contact  with  a 
blue  one  can  be  readily  perceived  as  isolated  ones  through  filtered  views.  The  same  scene 
will  be  easily  misinterpreted  by  conventional  binary  vision  system  as  single  object  because 
of  the  usage  of  blob  processing  and  run-length  encoding. 

It  is  easy  to  imagine  that  the  introduction  of  color  differentiations  for  current 
machine  vision  processing  will  greatly  extend  the  range  of  applications.  This  paper 
addresses  some  of  the  important  issues  of  color  filtering  and  binary  processing  as  a 
foundation  for  our  continual  effort  in  this  research. 
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